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Summary  For  the  past  six  years  the  Joint  Accreditation  Support  Activity  (JASA)  and  its  predecessor,  the  Susceptibility  Model 
Assessment  and  Range  Test  (SMART)  project,  have  provided  model  and  simulation  accreditation  support  to  US  Department  of 
Defense  (DoD)  acquisition  programs.  This  paper  summarizes  keys  to  cost  effective,  meaningful  accreditation  derived  from 
experience  working  with  missile,  aircraft,  and  electronic  warfare  systems  programs  in  various  stages  of  the  systems  acquisition 
cycle. 


1.  INTRODUCTION 

The  Susceptibility  Model  Assessment  and  Range  Test 
(SMART)  project  was  established  in  1992  by  the  Office 
of  the  Undersecretary  of  Defense  for  Acquisition  and 
Technology  with  three  goals:  to  develop  cost  effective 
model  and  simulation  (M&S)  verification,  validation 
and  accreditation  (W&A)  methodology,  to  demonstrate 
that  methodology  on  five  legacy  survivability  models,  and 
to  transition  the  methodology  and  the  lessons  learned  to 
the  larger  acquisition  community.  As  a  follow-on  to 
SMART,  the  Joint  Accreditation  Support  Activity 
(JASA)  was  instituted  in  1996  to  assist  acquisition 
programs  with  the  planning  and  implementation  of 
V&V  and  model  assessment  tasks  which  support 
accreditation. 

Several  trends  within  DoD  made  the  SMART  project  and 
JASA’s  services  timely.  The  overall  drawdown  in 
defense  spending  has  resulted  in  a  reduction  in  funding 
available  for  model  development  and  for  verification  and 
validation.  The  funding  drop  has  also  spurred  interest  in 
the  use  of  M&S  to  replace  or  supplement  more  expensive 
system  design,  test  and  evaluation,  and  training  methods. 
In  fact,  the  5000  series  of  directives  regarding  the 
acquisition  of  defense  systems  requires  the  use  of 
accredited  models  and  simulations  throughout  the 
acquisition  cycle  from  requirements  definition  to  post¬ 
deployment  support. 

As  the  pressure  to  use  modeling  and  simulation  has 
increased,  so  also  has  the  burden  on  program  managers 
to  defend  the  credibility  of  analysis  based  upon  the  results 
of  those  models.  As  used  in  the  5000  series  of  directives, 
accreditation  is  a  decision  that  a  model  or  simulation  is 


adequate  for  a  particular  purpose;  accreditation  decision 
authority  generally  rests  with  the  person  responsible  for 
the  use  of  the  model  output.  The  staff  of  the  SMART 
project  and  now  the  Joint  Accreditation  Support  Activity 
has  worked  closely  with  systems  acquisition  programs 
endeavoring  to  comply  with  the  5000  directives.  This 
paper  summarizes  the  major  lessons  learned  through 
SMART  and  JASA’s  experience  designing  and 
implementing  cost  effective  M&S  W&A  programs  to 
support  accreditation. 

2.  BASIC  PRINCIPLES 

First,  keep  in  mind  that  accreditation  is  a  decision.  The 
responsibility  of  the  accreditation  support  agent  is  to 
provide  the  accreditation  authority  with  sufficient 
information  to  make  an  intelligent  and  defensible 
decision  about  whether  or  not  a  model  or  simulation  is 
appropriate  for  their  needs.  The  amount  of  information 
necessary  to  defend  an  accreditation  decision  (and 
therefore  the  amount  of  resources  required  to  generate 
that  information)  is  dependent  upon  several  factors, 
including  the  importance  of  the  decisions  to  which  the 
model  contributes,  the  ramifications  of  a  wrong  decision 
based  upon  model  results  which  are  not  credible,  and  the 
dollar  amount  of  the  program.  Because  accreditation  of 
M&S  is  but  one  of  myriad  concerns,  program  managers 
are  motivated  to  spend  only  as  much  as  is  necessary  to 
establish  an  acceptable  level  of  confidence  in  model 
results.  The  question  often  becomes  “  How  confident  can 
I  afford  to  be?”  The  accreditation  support  agent  has  a 
responsibility  to  help  the  decision  authority  to  invest  his 
limited  accreditation  support  resources  on  those  tasks 
which  contribute  the  most  to  credibility  for  the  least  cost. 
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A  corollary  to  the  first  principle  is  that  accreditation  is 
both  a  technical  and  a  political  decision.  Insight  into  the 
proponents  and  detractors  of  the  model  and  the  influence 
they  exert  is  crucial.  As  DoD  increases  the  pressure  to 
reduce  the  number  of  models  in  use,  the  selection  of  a 
particular  model  by  high  visibility  acquisition  programs 
will  become  even  more  politically  charged.  A  well 
thought-out  and  well  articulated  rationale  for  selecting 
and  accrediting  a  particular  model  over  its  competitors  is 
critical  for  defending  the  decision  to  the  Defense 
Acquisition  Board  (DAB),  which  approves  transition 
from  one  phase  of  the  acquisition  process  to  another,  and 
to  Congress  who  ultimately  controls  the  funding.  From 
this  point  of  view,  accreditation  support  is  a  form  of  risk 
reduction  and  mitigation. 

The  second  idea  to  keep  in  mind  is  that  complying  with 
directives  such  as  the  5000  series  in  the  US  is  a 
secondary  goal.  The  primary  goal  of  accreditation  is  to 
understand  the  strengths  and  weaknesses  of  the  model 
under  consideration  and  their  effects  on  the  credibility  of 
analysis  based  on  model  output.  While  a  program 
manager’s  initial  interest  in  W&A  may  be  driven  by  the 
need  to  comply  with  directives,  the  long  term  goal  of  the 
program  is  to  make  decisions  which  are  well  grounded 
and  to  produce  a  quality  product.  The  accreditation 
support  agent  can .  be  helpful  in  providing  this 
perspective.  Since  W&A  must  be  addressed  to  check 
the  box  anyway,  let’s  make  it  meaningful. 

If  insight  into  the  credibility  of  M&S  output  is  the  goal  of 
meaningful  accreditation,  a  clear  statement  of  the 
intended  use  of  the  M&S  under  consideration  is  the 
foundation.  The  better  the  intended  use  is  defined,  the 
more  focused  (and  cost  effective)  the  V&V  and  model 
assessment  efforts  can  become.  In  the  tailored  W&A 
process  developed  by  SMART,  the  first  step  is  to 
understand  how  the  client  envisions  using  the  M&S. 
What  analysis  questions  are  being  addressed?  What 
measures  of  merit  will  be  used  to  quantify  answers  to 
those  questions?  What  level  of  fidelity  is  required  in  the 
answers?  What  assumptions  underlie  the  analysis? 

Once  those  issues  are  clarified,  model  acceptance  criteria 
can  be  developed  based  upon  the  intended  use.  The 
SMART  process  identifies  three  types  of  criteria: 
functional,  fidelity,  and  operational.  What  functions 
must  the  model  perform  in  order  to  address  the  questions 
of  interest?  To  what  level  of  detail  must  these  functions 
be  modeled,  and  what  level  of  accuracy  is  required  in 
model  output  to  meet  the  decision  maker’s  needs?  In 
what  kind  of  environment  must  the  M&S  run:  computer 
platform  and/or  operating  system,  real  time  vs.  non-  real 
time,  level  of  experience  or  training  of  model  operators. 


compatibility  with  other  analysis  efforts,  etc.  The 
acceptance  criteria  derived  from  the  answers  to  these 
questions  form  a  standard  against  which  the  model  of 
interest  is  assessed  for  accreditation,  and  V&V  efforts  are 
focused  on  those  features  of  the  M&S  and  those  portions 
of  the  code  which  are  related  to  those  criteria.  The  more 
the  problem  of  interest  can  be  defined,  the  less  effort  is 
wasted  on  examining  features  of  the  model  which  are  not 
relevant, 

3.  LESSONS  LEARNED 

While  clear  definition  of  analysis  goals  and  M&S 
implementation  strategy  is  a  key  to  cost  effective 
accreditation,  our  observation  has  been  that  most 
acquisition  programs  begin  with  only  a  loose 
understanding  of  how  analysis  and  M&S  will  be  used. 
Transforming  a  general  idea  of  the  types  of  issues  that 
need  addressing  and  a  vague  notion  of  how  M&S  might 
play  into  a  carefully  constructed  and  well  articulated  plan 
is  a  long  and  painful  process.  The  programs  to  which 
SMART/JASA  have  been  the  most  help  are  ones  in 
which  the  accreditation  support  personnel  got  involved 
with  the  analysis  team  early  and  stayed  involved  over 
time.  This  gives  the  accreditation  support  team  the 
opportunity  to  listen  carefully  to  the  analysis  team  to 
make  sure  they  understand  the  client’s  analysis  goals  (as 
well  as  the  simplifying  assumptions),  and  to  keep  up  as 
the  analysis  plans  naturally  evolve.  The  greater  the 
insight  into  the  analysis  effort,  the  more  focused  the 
accreditation  assessment  efforts.  Additionally,  the 
accreditation  support  team’s  knowledge  of  the  models 
and  analysis  areas  of  interest  often  prove  useful  in 
identifying  potential  risk  areas  early  enough  to  allow 
time  to  mitigate  or  avoid  the  problem.  Mitigation  might 
involve  making  model  improvements,  gathering  more 
credible  input  data,  or  using  different  tools  in  areas 
where  the  tool  of  choice  is  weak.  The  lesson  here  is  to 
get  involved  early  and  stay  involved. 

Another  basic  lesson  is  that  accreditation  is  an  iterative 
process.  As  circumstances  change  over  time,  the 
accreditation  issue  must  be  revisited.  Analysis  questions 
and/or  assumptions  change  from  one  phase  of  a  program 
to  another,  model  code  and  input  data  sets  change  on  a 
periodic  basis,  and  the  importance  of  M&S  in  the 
resolution  of  analysis  issues  shifts  from  phase  to  phase. 
Careful  documentation  of  previous  accreditation 
assessments  and  disciplined  configuration  management 
of  model  code  and  input  data  sets  on  the  part  of  both  the 
model  manager  and  the  program  office  analysis  staff 
minimizes  the  amount  of  effort  required  for  a 
reassessment.  Consistent  involvement  of  the 
accreditation  support  personnel  and  the  analysis  team 
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also  minimizes  the  learning  curve  for  each  successive 
round  of  accreditation. 

The  accreditation  support  staff  can  also  make  a 
substantial  contribution  to  the  programs  it  supports  by 
participating  in  the  process  by  which  they  decide  whether 
and  how  to  invest  program  funds  in  model  and  data 
improvement.  Because  program  managers  are  generally 
not  modeling  or  analysis  experts  themselves,  they  are 
vulnerable  to  persuasive  pleas  for  funding  for  model 
improvements  which  may  be  worthy,  but  are  not  high 
priority  given  the  analysis  needs  of  the  program.  The 
larger  and  more  visible  the  program,  the  more  appealing 
a  target  it  is  for  model  managers  and  model  users 
interested  in  model  improvements.  The  accreditation 
staff  can  be  of  help  by  consistently  asking  how  a 
proposed  model  improvement  effort  is  related  to  the 
analysis  needs  of  that  program  in  the  near  to  mid  term. 

Many  forces  converge  to  cause  program  office  personnel 
to  have  a  short  term  focus.  In  the  US,  program  office 
personnel  are  generally  military,  and  they  rotate 
assignments  every  two  to  three  years.  They  are  evaluated 
on  a  yearly  cycle,  and  they  are  barraged  with  issues 
which  are  pressing  today.  Accreditation  support  agents 
can  play  a  valuable  role  by  taking  a  longer  term,  more 
balanced  view.  Keeping  one  eye  on  the  long  term  brings 
two  benefits:  first,  it  allows  one  to  consider  how  to  make 
the  effort  currently  being  expended  of  most  use  in  the 
future,  and  it  encourages  one  to  look  ahead  to  the  impact 
of  current  modeling  limitations  on  later  phases  of 
analysis  in  time  to  implement  model  changes  or  select 
and  develop  proficiency  with  other  tools.  Limitations 
which  may  be  of  little  importance  to  the  questions  of 
interest  today  may  be  show  stoppers  for  questions  in  the 
next  phase.  The  other  eye  needs  to  be  firmly  fixed  on 
how  the  skills  and  knowledge  base  of  the  accreditation 
support  personnel  can  be  most  helpful  in  the  here  and 
now.  What  of  value  can  we  do  to  meet  today’s  needs 
with  the  resources  the  client  has  available  now? 

Program  managers  generally  do  not  have  a  modeling  or 
simulation  background,  and  accreditation  is  not  generally 
on  the  top  of  their  priority  list.  There  are  many  benefits 
to  having  a  central,  independent  activity  like  JASA 
which  keeps  current  on  policy  statements  and  the  way  in 
which  that  policy  is  being  implemented  (both  of  which 
are  in  great  flux  in  the  US),  has  experience  in  conducting 
verification,  validation  and  accreditation,  has  access  to 
expertise  in  various  modeling  areas,  has  active  contract 
vehicles  available  for  performing  W&A  related  tasks, 
and  maintains  a  repository  of  W&A  documentation. 
The  availability  of  expert  help  increases  the  probability 
that  programs  will  comply  with  W&A  policy  and  that 


their  efforts  will  be  meaningful.  Knowledge  of  what’s 
already  been  done  (and  documented!)  reduces  duplication 
of  effort  and  maximizes  the  usefulness  of  the  V&V  work 
funded  by  individual  programs  or  agencies  to  the  larger 
analysis  community.  The  availability  of  experienced 
personnel  also  reduces  the  cost  to  individual  programs  of 
designing  and  implementing  a  W&A  program.  As 
noted  earlier,  the  usefulness  of  an  accreditation  support 
agent  to  an  acquisition  program  is  also  dependent  upon 
the  willingness  to  listen  and  leant  about  the  particular 
needs  of  each  client. 

The  quality  and  appropriateness  of  the  model  code  itself 
is  only  half  of  the  problem.  Quality  of  model  input  data, 
its  appropriateness  to  the  model,  and  tracability  to 
authoritative  sources  is  the  other  half.  Data  quality  and 
pedigree  is  a  verification,  validation,  and  certification 
(W&C)  issue.  Regardless  of  the  quality  of  the  model 
code,  bad  input  leads  to  bad  output.  Increasing  emphasis 
within  DoD  on  use  of  officially  designated  authoritative 
sources,  in  concert  with  the  acknowledged  shortfalls  in 
some  areas  of  interest  create  a  great  challenge  for 
acquisition  programs.  Although  official  guidance  on 
W&C  of  data  is  currently  less  well  defined  than  that  on 
W&A  of  model  code,  data  quality  and  pedigree  and  the 
influence  on  the  credibility  of  model  output  must  be 
addressed  as  part  of  an  overall  accreditation  effort. 

Finally,  some  very  practical  observations.  First,  model 
usage  and  V&V  history  and  a  summary  of  assumptions, 
limitations,  and  errors  are  useful  to  a  broad  base  of  model 
users  and  they  are  two  of  the  least  expensive  V&V 
activities.  Usage  and  W&A  history  can  help  build  a 
case  for  provisional  or  preliminary  accreditation  based 
upon  community  acceptance,  and  the  assumptions  and 
limitations  can  help  a  potential  user  to  quickly  decide 
whether  a  model  deserves  consideration  for  a  particular 
problem.  Before  SMART,  these  topics  were  generally 
not  addressed  in  standard  model  documentation  sets. 
Second,  thick  skin  is  a  definite  asset  for  individuals 
involved  with  accreditation  support.  The  honest  broker 
is  often  not  everyone’s  favorite  individual. 

4.  CONCLUSION 

The  contribution  of  a  competent  and  proactive 
accreditation  support  team  can  extend  far  beyond 
providing  V&Vunderpiiuiings  for  meaningful 
accreditation  statements.  Experience  by  the 
SMART/JASA  team  suggests  several  keys  for  helping 
accreditation  support  personnel  make  the  most  of  their 
efforts.  Get  involved  early  in  the  program  and  stay 
involved.  Listen  and  learn.  Keep  one  eye  on  the  short 
term  and  the  other  on  the  big  picture.  Be  proactive  in 
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contributing  to  other  aspects  of  the  program  such  as  data 
and  model  improvement  investment  plans  and 
configuration  management  efforts.  Be  flexible.  Play  the 
role  of  an  honest  broker  even  though  this  is  often 


uncomfortable.  Have  a  thick  skin.  Above  all,  help  the 
customer  become  as  confident  in  M&S  results  as  he  or 
she  needs  to  be,  within  the  confines  of  the  time  and 
money  available. 


Visit  the  Joint  Accreditation  Support  Activity  at  our  web  site:  http://wwwjiawcwpns.navy.mil/-jasa 
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